Method and System for Facility Monitoring and Reporting to Improve Safety Using Robots

ABSTRACT

A system for facility monitoring and reporting to improve safety using one or more robots includes: a network; a plurality of autonomous mobile robots operating in a facility, the robots configured to monitor facility operation, the robots further configured to detect a predetermined critical condition, the robots operably connected to the network; a server operably connected to the robots over the network; and an individual robot operably connected to the server over the network, the individual robot operating in the facility, the robots not comprising the individual robot, the individual robot configured to monitor facility operation; wherein the robots are configured to regularly produce a regular report under normal operating conditions, the report displaying data received from the server, wherein the robots are further configured to produce to the server a critical condition report upon occurrence of the critical condition.

PRIORITY CLAIM

The present application claims the priority benefit of U.S. provisionalpatent application No. 62/948,440 filed Dec. 16, 2019 and entitled“Method and System for Facility Monitoring and Reporting to ImproveSafety Using Robot Swarms on High Alert,” the disclosure of which isincorporated herein by reference.

SUMMARY

Embodiments of this invention relate in general to a system and methodfor facility monitoring and reporting to improve safety using one ormore robots. More specifically, embodiments of this invention relate toa system and method for facility monitoring and reporting to improvesafety using one or more robots configured to go to be used by one ormore of a robot, a self-driving automobile, and another autonomousagent.

For example, according to embodiments of the invention, a robot goes onhigh alert upon discovery of a critical condition.

Embodiments of the invention relate in general to a system and methodfor facility monitoring and reporting to improve safety using one ormore robots.

A system for facility monitoring and reporting to improve safety usingone or more robots, includes: a network, a plurality of autonomousmobile robots operating in a facility, the robots configured to monitorfacility operation, the robots further configured to detect apredetermined critical condition, the robots operably connected to thenetwork; a server operably connected to the robots over the network; anda user interface (UI), the UI operably connected to the server, the UIconfigured to select data received from the server, the UI furtherconfigured to display pertinent data to a user, wherein the robots areconfigured to regularly produce a regular report under normal operatingconditions, the report displaying data received from the server, whereinthe robots are further configured to produce to the server a criticalcondition report upon occurrence of the critical condition.

A method for detecting for facility monitoring and reporting to improvesafety using one or more robots includes: using a system comprising anetwork, the system further comprising a plurality of robots operatingin a facility, the robots configured to monitor facility operation, therobots further configured to detect a critical condition, wherein therobots are further configured to go into high alert upon detection ofthe critical condition, the robots operably connected to the network,the system further comprising a server operably connected to the robotsover the network, wherein the robots are configured to regularly produceto the server a regular report under normal operating conditions,wherein the robots are further configured to produce to the server acritical condition report upon occurrence of the critical condition,receiving, by the server, from a detecting robot, a critical conditiondetected by the detecting robot; switching the detecting robot, by theserver, from periodic, lower-priority data recording, to high-priority,real-time, continuous data recording and direct uploading; sending thecritical condition, by the server, to one or more other robots; andrequesting assistance, by the server, for the detecting robot, from theother robots.

A method for detecting for facility monitoring and reporting to improvesafety using one or more robots includes: using a system comprising anetwork, the system further comprising a plurality of robots operatingin a facility, the robots operably connected to the network, the robotsconfigured to monitor facility operation, the robots further configuredto detect a critical condition, wherein the robots are furtherconfigured to go into high alert upon detection of the criticalcondition, the system further comprising a server operably connected tothe robots over the network, wherein the robots are configured toregularly produce to the server a regular report under normal operatingconditions, wherein the robots are further configured to produce to theserver a critical condition report upon occurrence of the criticalcondition, receiving, by the server, from a detecting robot, a criticalcondition detected by the detecting robot; requesting data regarding thecritical condition, by the server, from a robot; displaying, by theserver, on a user interface (UI) visible to a user, the UI operablyconnected to the server, the critical condition; generating, by theserver, a heat map of the critical condition; generating, by the server,a path usable by the robots around the critical condition; switching thedetecting robot, by the server, from periodic, lower-priority datarecording, to high-priority, real-time, continuous data recording anddirect uploading; sending the critical condition, by the server, to oneor more other robots; requesting assistance, by the server, for thedetecting robot, from the other robots; receiving, by the server, fromthe determining robot, a determination that one or more of the criticalcondition and a number of robots near the critical condition isproblematic; sending, by the server, to one or more other robots, acommand blocking the one or more other robots from approaching thecritical condition; receiving, by the server, from a user, aninstruction regarding preventive measures to resolve the criticalcondition; implementing, by the server, the user's instruction regardingthe preventive measures to resolve the critical condition; sending, bythe server, to the UI, for display to the user, a robot selection panelcomprising a plurality of robots usable to approach the criticalcondition; receiving, by the server, from the UI, a user selection of arobot from the robot selection panel; and transmitting, by the server,to each of the user-selected robots, a respective instructioninstructing each of the user-selected robots to approach the criticalcondition.

DESCRIPTION OF THE DRAWINGS

The accompanying drawings provide visual representations which will beused to more fully describe various representative embodiments and canbe used by those skilled in the art to better understand therepresentative embodiments disclosed herein and their inherentadvantages. In these drawings, like reference numerals identifycorresponding elements.

FIGS. 1A-1C are a set of three drawings of a system for facilitymonitoring and reporting to improve safety using one or more robots.

FIGS. 2A-2F are a set of six drawings of a system for facilitymonitoring and reporting to improve safety using one or more robots.

FIG. 3 is a flow chart illustrating a method for detecting a criticalcondition by an autonomous mobile robot using a system for facilitymonitoring and reporting to improve safety using one or more robots.

FIG. 4 is a flow chart illustrating a method for detecting a criticalcondition by an autonomous mobile robot using a system for facilitymonitoring and reporting to improve safety using one or more robots.

FIGS. 5A-5F are a set of diagrams illustrating a method for detecting acritical condition by an autonomous mobile robot using a system forfacility monitoring and reporting to improve safety using one or morerobots.

FIGS. 6A-6O are a set of diagrams illustrating a method for detecting acritical condition by an autonomous mobile robot using a system forfacility monitoring and reporting to improve safety using one or morerobots.

DETAILED DESCRIPTION

While the present invention is susceptible of embodiment in manydifferent forms, there is shown in the drawings and will herein bedescribed in detail one or more specific embodiments, with theunderstanding that the present disclosure is to be considered asexemplary of the principles of the invention and not intended to limitthe invention to the specific embodiments shown and described. In thefollowing description and in the several figures of the drawings, likereference numerals are used to describe the same, similar orcorresponding parts in the several views of the drawings.

Embodiments of the invention relate in general to a system and methodfor facility monitoring and reporting to improve safety using one ormore robots. For example, the facility comprises a warehouse.

According to embodiments of the invention, robots can be used to improvesafety of the facility by tracking speed of one or more vehicles movingwithin the facility. For example, the vehicles comprise one or more offorklifts, golf carts, autonomous guided vehicles (AGVs), autonomousmobile robots (AMRs), and other vehicles. According to yet otherembodiments of the invention, robots can detect a velocity of objects inthe vicinity using sensor data comprising one or more of a laser scan, acamera point cloud, radio frequency identification (RFID) data, inertialmeasurement unit (IMU) data, and other data. Robots can produceinformation on safety and security incidents such as, for example, nearmisses and a number of vehicles that exceed the speed limit. The robotscan produce such reports with any desired frequency, for example with afrequency that is one or more of daily, weekly, and monthly.

For example, the robot further comprises a critical condition detectorconfigured to monitor for occurrence of a critical condition. Forexample, the critical condition is one or more of predetermined anddefined by a user. For example, the critical condition comprises one ormore of a critical condition comprises ed as one or more of a flood; anear miss of two or more entities; a vehicle exceeding a speed limit;traffic congestion; a spill; a blocked emergency exit; a fire;unexpected sprinkler activation, water on the floor from a roof leak; aheavy machinery accident; a robotic detection of one or more of amovement out of permissible bounds, a collision, or a significantlydamaged surface over which the robot is travelling; a robot receiving anexternal trigger from the server to record data in a specific area ofinterest; a robot receiving an external trigger from the server torecord data regarding a specific object of interest; exceeding one ormore of a predetermined numerical threshold and a predeterminedproximity threshold for a number of humans, robots, and objects of aspecified category; passage of a predetermined time period from abeginning of a task; and absence of an object from a location where thesystem expects its presence.

The robot further comprises an alarm module configured to take actionupon receipt of a trigger notifying the alarm module that the criticalcondition has occurred. If the critical condition detector detects acritical condition, the robot triggers the alarm module. Upon receivingthe trigger, the alarm module performs one or more of sending a stopcommand to the server stream, sending an error code to the robots, andstreaming to the cloud the data describing the critical condition. Thestop command orders the server to stop the critical condition. The errorcode corresponds to the critical condition.

According to other embodiments of the invention, robots can be used tosense one or more of a region of a facility and an entire facility toimprove safety of the facility by detecting near misses when two or moreentities in the facility come closer to each other than a predeterminedpermissible distance. For example, the entities comprise one or more ofa human, a vehicle, a forklift, a golf cart, an autonomous guidedvehicle, a robot, and a rack.

For example, the permissible distance comprises one foot. For example,the permissible distance comprises one inch. For example, the robot isconfigured to send an alert upon occurrence of a near miss. For example,the alert comprises a recording of the near miss. For example, therecording comprises a video recording. For example, the robot sends thealert including the video recording to a manager of the facility.

For example, according to yet further embodiments of the invention, if arobot comes too close to an obstacle, it triggers a nearby camera tostart monitoring the area surrounding the robot. According to stillother embodiments of the invention, the system also responds by sendinga code to the offending robot over a communication network, the codebeing configured to trigger other robots to respond to the near miss.For example, the network comprises one or more of Bluetooth, WiFi, thecloud, and another network. The responding robot can collect additionaldata regarding precipitating factors behind the near miss that would nototherwise be collected. For example, the responding robot startsrecording data using its sensor. According to still other embodiments ofthe invention, the responding robot sends a code over the network to atleast one other robot in the vicinity, the code being configured totrigger other robots to respond to the near miss.

According to other embodiments of the invention, robots continuouslydetect and monitor a number of one or more of humans, forklifts, golfcarts, autonomous guided vehicles, autonomous mobile robots and othermoving obstacles in the facility. This information from multiple robotsis sent to the system over the network. For example, the system uses theinformation to generate statistics on facility workflow over time. Forexample, the system uses the information to generate a lane planningalgorithm providing a more optimal route for the robots.

According to yet further embodiments of the invention, robots can beused to sense one or more of a region of a facility and an entirefacility to improve safety of the facility by doing one or more ofdetecting congestion and reporting the congestion. Congestion may beloosely defined as a condition in which more than an optimum number ofmoving entities are positioned within a given area of the facility. Forexample, congestion may be defined as occurring whenever more than sixmoving entities, either humans or vehicles, occupy an area of twentysquare feet. For example, congestion may be defined as occurringwhenever more than three moving entities, either humans or movingvehicles, occupy an area of one square meter.

For example, the robot is configured to send an alert upon occurrence ofcongestion exceeding a specified congestion limit. For example, therobot sends the congestion alert to a manager of the facility.

According to still other embodiments of the invention, robots can beused to sense one or more of a region of a facility and an entirefacility to improve safety of the facility by checking one or morepoints of interest. For example, the robots check the one or more pointsof interest to check for one or more or presence of a desired conditionand absence of a non-desired condition. For example, the non-desiredcondition comprises a critical condition.

For example, in the event that the point of interest comprises a fireextinguisher, a desired condition is functionality of the fireextinguisher and a non-desired condition is non-functionality of thefire extinguisher. For example, the robot may determine that the fireextinguisher is non-functional it is one or more of broken and missingan adequate level of fire extinguishing fluid in its canister. In thecase of the fire extinguisher, according to further embodiments of theinvention, the robot is configured to replace the non-functional fireextinguisher with a functional replacement fire extinguisher.

The system is configured to conduct automated building safety audits.The safety audit investigates a status of one or more of an emergencyexit, a fire extinguisher condition, a fire extinguisher placement,access to an electrical panel, status of a rack protector, status of aguard rail, and another condition of interest. The system is furtherconfigured to receive real-time alerts in advance of mandatoryinspections by identifying one or more of a blocked emergency exit, afire extinguisher condition, a fire extinguisher placement, access to anelectrical panel, a damaged rack protector, a damaged guard rail, andanother condition of interest.

If a robot is passing in the vicinity of one or more of an emergencyexit and a fire extinguisher that has not been checked within auser-specified amount of time, the passing robot can be assigned tomonitor the one or more of an emergency exit and a fire extinguishereven though they are not exactly on the route.

For example, in the event that the point of interest comprises anemergency exit, a desired condition is the emergency exit being closedand a non-desired condition is the emergency exit being open. In thecase of the emergency exit, according to further embodiments of theinvention, the robot is configured to close an open emergency exit.

According to yet other embodiments of the invention, robots can be usedto sense one or more of a region of a facility and an entire facility tomonitoring work areas. For example, the robots monitor the facility workareas using facility safety standards, for example, “Five Steps” (5S).

The robot detects whether one or more of a box and a pallet is within adesignated area to ensure facility compliance with 5S regulations.

A feature offered by embodiments of the invention is an instant hazardavoidance zone according to which, from anywhere on any device, thesystem can instantly isolate robots away from an emergency incident suchas one or more of an unexpected sprinkler activation, water on the floorfrom a roof leak, and a heavy machinery accident.

Another feature offered by embodiments of the invention is proactivelyre-slotting robot shifts ahead of one or more of increased customerdemands and seasonal shifts in work patterns. The system synchronizesstorage locations with one or more of movement of goods and facilitytraffic conditions. Onboard robotic sensors capture areas of heavycongestion. Additionally, an automatic RFID tag onboard the robot doesone or more of track and monitor stock-keeping unit (SKU) velocity. Thesystem then uploads the data using the network to optimize facilityspace utilization.

One application of the system and method for facility monitoring andreporting to improve safety using one or more robots is marking the mapof the facility with one or more areas dictating one or more of aminimum speed limit and a maximum speed limit for a vehicle. Forexample, one or more of the minimum speed limit and the maximum speedlimit may be the same for all vehicles. Alternatively, or additionally,one or more of the minimum speed limit and the maximum speed limitapplies to a specified vehicle type only. According to furtherembodiments of the invention, the robot is configured to send an alertupon detecting a vehicle of the specified type exceeding the designatedspeed limit. For example, the robot sends the alert to a manager of thefacility.

Another application of the system and method for facility monitoring andreporting to improve safety using one or more robots is creating a heatmap viewable by a user, the heat map visually showing a congestion levelin the facility. Alternatively, or additionally, the server generates aheat map of the critical condition. For example, the heat map shows thecongestion level throughout the facility. For example, the heat mapshows the congestion level throughout the facility for one or more of avehicle and a human. For example, the heat map shows the congestionlevel in the facility for a designated type of vehicle. For example, thevehicle type comprises one or more of forklift, golf cart, autonomousguided vehicle, autonomous mobile robot, and another vehicle type. Forexample, after generating the heat map, the server sends the heat map tothe UI for display.

The present invention concerns multiple robots performing regularmonitoring of their environment, when a critical condition is noticed.In one embodiment, the critical condition is noticed by a user who hasbeen monitoring the collective data of the robots. In anotherembodiment, the critical condition is noticed by a robot, which notifiesits robot colleagues of the condition, as well as notifying a server ofthe condition. Once the critical condition has been noticed andreported, a number of different responses can result according toembodiments of the invention. One such response involves the systemplacing the robots in response to the condition. The present inventionincludes a UI for performing one or more of monitoring the robots andmanaging the robots.

According to embodiments of the invention, the robot can be programmedto detect one or more of a range of critical conditions. One criticalcondition known as “IMU/Odom” (Inertial Measurement Unit/Odometrysensors) occurs if a robot detects one or more of a movement out ofpermissible bounds, a collision, or a significantly damaged surface overwhich the robot is travelling.

A second critical condition known as “external triggers to record datain a specific area” comprises a notification received by the robotindicating that the area in which the robot is located is of interest.For example, the notification indicates that the travel area is ofinterest to one or more of a user and a second robot that the systeminstructs to assist in responding to the critical condition. Forexample, the system may instruct the second robot to assist inresponding to the critical condition if the first robot is in a positionwhere a collision may occur. For example, the system may instruct thesecond robot to assist in responding to the critical condition if thesystem recognizes an object that recently arrived in the area.

The system triggers this critical condition when the robot's internallocalization data indicates that the robot is within the area ofinterest.

A third critical condition known as “human/forklift/conveyer detection(triggered in a 1 m radius vicinity)” is similar to the second criticalcondition, except that now a particular object of interest is beingdetected, rather than a particular area. The system triggers thiscritical condition when the robot's sensor detects the object ofinterest, and the internal localization data indicates that the robot iswithin a specified distance of the object. The robot sends trigger codesover the network configured to instruct other nearby robots to pausetheir current activity and prioritize the triggered task.

A fourth critical condition known as “too manyhumans/robots/forklifts/carts” is triggered by the system when one ormore of a predetermined numerical threshold and a predeterminedproximity threshold is reached. The robots send this informationimmediately to the cloud. The system then does one or more of instructother robots in the vicinity to record sensor data in the specific areaand instruct idle robots to monitor the area of interest.

A fifth critical condition known as “keep track of any nav times andnotify if all of sudden tasks took too long” is triggered by the systemwhen a predetermined time period passes from the beginning of a task.

A sixth critical condition known as “shelf/marker missing” is triggeredby the system when an object that the system expects will be present isinstead missing.

A user utilizes a user interface (UI) to manage the robots. The robotsgenerate data that the robots send to the server. The robots send thedata to the server one or more of periodically and continuously. Forexample, the robots send the data to the server periodically duringnormal operation. For example, the robots send the data continuouslyupon detection of a critical condition. The system uses the receiveddata to generate a representation of the system displayable on the UI.For example, the displayed data comprises one or more of the regularreport and the critical condition report. For example, the displayeddata comprises the critical condition.

For example, the UI comprises a first map displaying a location of arobot and a confidence associated with the detected critical condition.The confidence comprises a cumulative confidence that the robots have inan accuracy of the identified critical condition. The first map enablesa user to determine a probability that the critical condition hasoccurred.

For example, the UI comprises a second map displaying a location of adetected critical condition. The second map further comprises a relativeposition of at least one robot. The second map enables the user to doone or more of monitor and manage progress of the robots in respondingto the critical condition.

For example, the UI further comprises a timeline comprising a view ofthe data over time. For example, the timeline comprises one or more of aphotograph, an audio recording, and a video recording. For example, thetimeline presents the data in chronological order. The timeline enablesa user to investigate possible causes of a detected critical condition.

The UI comprises a configuration button configured upon selection toopen a settings page displaying settings options for the user. Thesettings comprises one or more of notifications settings, filteringsettings, and visualization settings.

Notifications settings enable the user to specify details of systemnotifications provided to the user. For example, the notificationssettings comprises one or more of 1) a means of notification (forexample, text messages, electronic mail, telephone call, and the like),2) a type of notification (for example, Robot Operating System (“RViz”),small UI messages (“toasts”) at the bottom of the UI screen, and thelike), and 3) a subject of the notifications (for example, mapaugmentations, robot localization, and the like).

Filtering settings enable the user to only receive data of interest tothe user and to filter out other data. For example, the filteringsettings comprise one or more of type of event, time of event, andpriority of event. Filtering settings can be implemented using one ormore of a heat map configured to overlay data on the map and a scrubbingfunctionality that allows the user to scroll through the timeline whilewatching a visualization of the data.

Visualization settings enable the user to choose a kind of content todisplay in the UI. Examples of such content include new annotations andnav mesh. For example, the new annotation comprises one or more of aforklift and a package. For example, the nav mesh comprises one or moreof an obstruction and a slippage. Choosing one or more of these contenttypes causes the chosen content to appear on the UI, in one or more ofthe first map, the second map, and the timeline. These settings providethe user with a fuller picture of the area of interest.

The visualization settings also allow the user to select one or moredetectors. Each detector may be configured to identify one or morecritical conditions. After selecting one or more detectors, the user canuse one or more of the first map, the second map, the timeline, andfiltering settings to do one or more of analyze critical conditions andreview underlying sensor data.

The first map allows the user to create annotations that specify safetyrules. For example, the safety rule comprises a requirement that an areamust be kept free, for example, the space in front of an emergency exit.For example, the safety rule comprises a maximum speed limit for anarea. The UI further comprises a visualization usable to highlight acritical condition detected by the robots.

FIGS. 1A-1C are a set of three drawings of a system for facilitymonitoring and reporting to improve safety using one or more robots.

FIG. 1A is a drawing of a system 100A for facility monitoring andreporting to improve safety using one or more robots 110. The system 100further comprises a cloud 120 that is operably connected over a networkconnection 125 to the robot 110. The robot 110 is configured to switchto high alert upon detection by the system 100 of certain conditions.

For example, the robot 110 comprises one or more sensors 130. Forexample, the sensor 130 comprises one or more of a depth camera, atwo-dimensional (2D) light detection and ranging (LIDAR) sensor, andanother sensor. While the robot 110 is navigating around the facility(not shown), the sensors 130 are generating data 140. For example, thedata 140 comprises raw data. For example, the data 140 comprises one ormore of video data, audio data, location data, point cloud data, andother data. Preferably, but not necessarily, the sensors 130 are runningcontinuously 150. The sensors 130 are continuously transmitting the data140 to the cloud 120 via the network connection 125.

For example, the robot 110 uses a server stream module (not shown) toupload the data 140 to the cloud 120. Although not complete orcontinuous, the periodic data 140 streaming from the robot 110,optionally combined with periodic data 140 streaming from other robots(not shown), is generally sufficient to provide the system 100 with amulti-dimensional view of the facility (not shown). The streamed data,presented to a user through the UI, enables the user to detect andinvestigate problems.

The cloud 120 comprises data storage 160, the data storage 160configured to store the data 140 received over the network connection125 from the robot 110. The data storage 160 is operably connected toone or more processors 170A and 170B. For example, the processors 170Aand 170B are configured to generate metrics using the data 140 receivedfrom the robot 110. For example, the metrics comprise one or more of astatistical algorithm and a machine learning algorithm. For example, themetrics comprise one or more of an average, a median, a mode, avariance, a linear regression, a k mean clustering, and the like.

The processors 170A and 170B are operably connected to a user interface(UI) 175. Output from the processors 170A and 170B can be visualized fora user in the UI. For example, one or more of a map, a criticalcondition and another feature of interest can be visualized for the userin the UI.

FIG. 1B is a drawing of a system 100B for facility monitoring andreporting to improve safety using one or more robots 110. The system 100again further comprises the cloud 120 that is operably connected overthe network connection 125 to the robot 110. For example, the cloud 120again comprises the cloud 120. The robot 110 is again configured toswitch to high alert upon detection by the system 100 of certainconditions.

For example, the robot 110 again comprises one or more sensors 130. Forexample, the sensor 130 again comprises one or more of a depth camera, atwo-dimensional (2D) light detection and ranging (LIDAR) sensor, andanother sensor. While the robot 110 is navigating around the facility(not shown), the sensors 130 again are generating data 140. For example,the data 140 comprises raw data. For example, the data 140 againcomprises one or more of video data, audio data, location data, pointcloud data, and other data. Preferably, but not necessarily, the sensors130 are again running continuously 150.

The robot 110 further comprises a high priority error detection module177 configured to detect errors having high priority in the data 140.The high priority is determined based on one or more of user input and apredetermined priority. The sensors 150 are operably connected to thehigh priority error detection module 177. The sensors 130 are againcontinuously transmitting the data 140 to the cloud 120 via the networkconnection 125.

For example, the robot 110 again uses a server stream module (not shown)to upload the data 140 to the cloud 120. Although not complete orcontinuous, the periodic data 140 streaming from the robot 110,optionally combined with periodic data 140 streaming from other robots(not shown), is generally sufficient to provide the system 100 with amulti-dimensional view of the facility (not shown). The streamed data,presented to a user (not shown) through the UI 175, enables the user(not shown) to detect and investigate problems. The robot 110 furthercomprises a data recording module 180 configured to record the data 140.

The cloud 120 again comprises data storage 160, the data storage 160again configured to store the data 140 received over the networkconnection 125 from the robot 110. The data storage 160 is againoperably connected to one or more processors 170A and 170B. For example,the processors 170A and 170B are configured to generate metrics usingthe data 140 received from the robot 110. For example, the metricscomprise one or more of a statistical algorithm and a machine learningalgorithm. For example, the metrics comprise one or more of an average,a median, a mode, a variance, a linear regression, a k mean clustering,and the like. The processors 170A and 170B are again operably connectedto a user interface (UI) 175 configured to visualize for the user (notshown) one or more of a map and a critical condition and another featureof interest.

The error detection module 177 is operably connected to one or moreother robots 182A, 182B, and 182C. As depicted, the error detectionmodule 177 is connected to three other robots 182A, 182B, and 182C. Ifthe high priority error detection module 177 detects one or more errorsin the data 140 having the high priority based on the one or more ofuser input and a predetermined priority, then the error detection module177 transmits error codes 185 describing the detected high priorityerror in the data 140 to the one or more other robots 182A, 182B, 182C.

FIG. 1C is a drawing of a steady-state system 100C for facilitymonitoring and reporting to improve safety using one or more robots 110.The system 100 again further comprises the cloud 120 that is operablyconnected over the network connection 125 to the robot 110. For example,the cloud 120 again comprises the cloud 120. The robot 110 is againconfigured to switch to high alert upon detection by the system 100 ofcertain conditions.

For example, the robot 110 receives data 140. For example, the data 140again comprises one or more of video data, audio data, location data,point cloud data, and other data. For example, the data 140 comprisesrecorded data. The robot 110 further comprises a data analyzer 188configured to analyze the data. The data analyzer 188 analyzes the data140 while the robot 110 is charging.

The robot 110 transmits the data 140 to the data analyzer 188. Afteranalyzing the data 140, the data analyzer 188 outputs metrics 190. Forexample, the metrics comprise one or more of a statistical algorithm anda machine learning algorithm. For example, the metrics comprise one ormore of an average, a median, a mode, a variance, a linear regression, ak mean clustering, and the like. For example, the data analyzer 188comprises one or more long-running analytics processes. The robot 110transmits the metrics 190 to the cloud 120 via the network connection125. For example, the robot 110 uploads the metrics 190 to the cloud 120via the network connection 125.

The robot 110 further comprises a high priority error detection module177 configured to detect errors having high priority in the data 140.The high priority is determined based on one or more of user input and apredetermined priority. The sensors 150 are operably connected to thehigh priority error detection module 177. The sensors 130 are againcontinuously transmitting the data 140 to the cloud 120 via the networkconnection 125.

For example, the robot 110 again uses a server stream module (not shown)to upload the data 140 to the cloud 120. The streamed data, presented toa user (not shown) through the UI (not shown), enables the user (notshown) to detect and investigate problems. The robot 110 furthercomprises a data recording module 180 configured to record the data 140.

The cloud 120 again comprises data storage 160, the data storage 160again configured to store the data 140 received over the networkconnection 125 from the robot 110. For example, the data storage 160comprises multi-tier cloud-based data storage 160. The data storage 160is again operably connected to an processor 170. For example, theprocessor 170 again are configured to generate metrics using the data140 received from the robot 110.

FIGS. 2A-2F are a set of six drawings of a system for facilitymonitoring and reporting to improve safety using one or more robots.

FIG. 2A is a drawing of a system 200A for facility monitoring andreporting to improve safety comprising one or more robots 110A and 110B.As depicted, the system comprises a first robot 110A and a second robot110B. The forklift 210 is moving in a direction indicated by velocityarrow 215. The first robot 110A comprises a first sensor 220A. Asdepicted, the first sensor 220A comprises a first sensor 220A. The firstrobot 110A detects the forklift 210 using the first sensor 220A. Thefirst sensor 220A detects the forklift 210 using first robot detectionbeams 230A. The second robot comprises a second sensor 220B. Asdepicted, the second sensor 220B comprises a second sensor 220B. Thesecond robot 110B detects the forklift 210 using the second sensor 220B.The second sensor 220B detects the forklift using second robot detectionbeams 230B. The system 200A further comprises a human worker 240.

FIG. 2B is a drawing of a system 200B for facility monitoring andreporting to improve safety comprising one or more robots 110, showing arobot 110 detecting both a first human 240A and a moving forklift 210.The robot 110 again comprises the sensor 220. As depicted, the system200B comprises one robot 110. The system 200B comprises the forklift210. The system 200B further comprises a first human worker 240A and asecond human worker 240B. The forklift 210 is moving in a directionindicated by the velocity arrow 215. The robot 110 detects the firsthuman 240A using the sensor 220. The robot 110 also detects the forklift210 using the sensor 220. The sensor 220 detects the first human 240Ausing a first robot detection beam 230A. The sensor 220 detects theforklift 210 using a second robot detection beam 230B. The sensor 220detects a near miss of the moving forklift 210 and the first humanworker 240A. The near miss activates a critical condition for the robots110, to which the robots 110 respond by going into high alert.

FIG. 2C is a drawing of a system 200C for facility monitoring andreporting to improve safety comprising one or more robots 110. The robot110 again comprises the sensor 220. As depicted, the system 200Ccomprises one robot 110. The system again comprises the forklift 210.The system 200C further comprises human workers 240A-240D. The forklift210 is moving in a direction indicated by the velocity arrow 215. Therobot 110 detects the first human 240A using the sensor 220. The robot110 also detects the second human 240B using the sensor. The sensor 220detects the first human 240A using a first robot detection beam 230A.The sensor 220 detects the second human 240B using a second robotdetection beam 230B. The sensor 220 detects a near miss of the movingforklift 210 and the first human worker 240A. The near miss activates acritical condition for the robots 110, to which the robots 110 respondby going into high alert.

FIG. 2D is a drawing of a system 200D for facility monitoring andreporting to improve safety comprising one or more robots 110, showingthe robot 110 checking safety of a facility 250. As depicted, the system200D comprises one robot 110. The facility 250 comprises a fireextinguisher 260 and an emergency exit 270. As depicted, the emergencyexit 270 comprises a fire door 270. The robot 110 again comprises thesensor 220. The robot 110 detects the fire extinguisher 260 using thesensor 220. The robot 110 also detects the emergency exit 270 using thesensor. The sensor 220 detects a left side 277A of the emergency exit270 using a first robot detection beam 230A. The sensor 220 detects thefire extinguisher 260 using one or more of a second robot detection beam230B, a third robot detection beam 230C, a fourth robot detection beam230D, and a fifth robot detection beam 230E. The sensor 220 detects aright side 277B of the emergency exit 270 using a sixth robot detectionbeam 230F. Based on the results of one or more of the first robotdetection beam 230A and the sixth robot detection beam 230F, the robot110, in communication with the server (not shown), determines apossibility that a critical condition exists involving blockage of theemergency exit 270. The robot 110 receives from the server (not shown)instructions to further investigate the possible blockage of theemergency exit 270. The robot 110 therefore detects the emergency exit270 using a seventh robot detection beam 230G. The seventh robotdetection beam 230G detects a box 275 blocking the emergency exit 270.The blocked emergency exit 270 activates a critical condition for therobots 110, to which the robots 110 respond by going into high alert.

FIG. 2E is a drawing of a system 200E for facility monitoring andreporting to improve safety comprising one or more robots 110, showingthe robot 110 checking safety of a facility 250. The facility 250comprises a designated pallet location 279. As depicted, the designatedpallet location 279 comprises a pallet 280. As depicted, the pallet 280comprises a payload 281. As depicted, the payload 281 comprises a box281. The robot 110 again comprises the sensor 220. The robot 110 detectsthe pallet 280 using the sensor 220. The robot 110 also detects thepayload 281 using the sensor. The sensor 220 detects the pallet 280using a first robot detection beam 230A. The sensor 220 detects thepayload 281 using a second robot detection beam 230B. Based on theresults of one or more of the first robot detection beam 230A and thesixth robot detection beam 230F, the robot 110, in communication withthe server (not shown), determines that one or more of the pallet 280and the payload 281 are properly located entirely within the designatedpallet location 279. As depicted, both the pallet 280 and the payload281 are properly located entirely within the designated pallet location279. The proper location of the one or more of the pallet 280 and thepayload 280 therefore does not activate a critical condition for therobots 110.

The robot 110 also checks one or more of an orientation of the pallet280 and a size of the payload 281 to ensure that no overhanging objectsoutside the pallet boundary violate 5S regulations, thereby creating acritical condition for the robot 110, to which the robot 110 responds bygoing into high alert.

FIG. 2F is a drawing of a system 200F for facility monitoring andreporting to improve safety comprising one or more robots 110, showingthe robot 110 checking safety of a facility 250. The facility 250comprises a designated pallet location 279. As depicted, the designatedpallet location 279 comprises a pallet 280. As depicted, the pallet 280comprises a payload 281. As depicted, the payload 281 comprises a box281. The robot 110 again comprises the sensor 220. The robot 110 detectsthe pallet 280 using the sensor 220. The robot 110 also detects thepayload 281 using the sensor. The sensor 220 detects the pallet 280using a first robot detection beam 230A. The sensor 220 detects thepayload 281 using a second robot detection beam 230B. Based on theresults of one or more of the first robot detection beam 230A and thesixth robot detection beam 230F, the robot 110, in communication withthe server (not shown), determines that one or more of the pallet 280and the payload 281 are not properly located entirely within thedesignated pallet location 279. As depicted, both the pallet 280 and thepayload 281 are not properly located entirely within the designatedpallet location 279. The improper location of the one or more of thepallet 280 and the payload 280 therefore activates a critical conditionfor the robot 110, to which the robot 110 responds by going into highalert.

The robot 110 also checks one or more of an orientation of the pallet280 and a size of the payload 281 and determines that overhangingobjects outside the pallet boundary violate 5S regulations, therebycreating a critical condition for the robot 110, to which the robot 110responds by going into high alert.

FIG. 3 is a flow chart illustrating a method 300 for detecting acritical condition by an autonomous mobile robot using a system forfacility monitoring and reporting to improve safety using one or morerobots.

The order of the steps in the method 300 is not constrained to thatshown in FIG. 3 or described in the following discussion. Several of thesteps could occur in a different order without affecting the finalresult.

In step 310, using a system comprising a network, the system furthercomprising a plurality of robots operating in a facility, the robotsconfigured to monitor facility operation, the robots further configuredto detect a critical condition, the robots operably connected to thenetwork, the system further comprising a server operably connected tothe robots over the network, wherein the robots are configured toregularly produce to the server a regular report under normal operatingconditions, wherein the robots are further configured to produce to theserver a critical condition report upon occurrence of the criticalcondition, the server receives, from a detecting robot, a criticalcondition detected by the detecting robot. For example, the criticalcondition is identified as occurring in a critical area. Block 310 thentransfers control to block 320.

In step 320, the detecting robot switches from periodic, lower-prioritydata recording, to high-priority, real-time, continuous data recordingand direct uploading. Block 320 then transfers control to block 330.

In step 330, the server sends the critical condition to one or moreother robots. For example, the aid comprises one or more of providingfurther evidence of the critical condition and helping resolve thecritical condition. Preferably, but not necessarily, the other robotsare located in the critical area. Block 330 then transfers control toblock 340.

In step 340, the server requests assistance from the other robots. Block340 then terminates the process.

FIG. 4 is a flow chart illustrating a method 400 for detecting acritical condition by an autonomous mobile robot using a system forfacility monitoring and reporting to improve safety using one or morerobots.

The order of the steps in the method 400 is not constrained to thatshown in FIG. 4 or described in the following discussion. Several of thesteps could occur in a different order without affecting the finalresult.

In step 410, using a system comprising a network, the system furthercomprising a plurality of robots operating in a facility, the robotsoperably connected to the network, the robots configured to monitorfacility operation, the robots further configured to detect a criticalcondition, wherein the robots are further configured to go into highalert upon detection of the critical condition, the system furthercomprising a server operably connected to the robots over the network,wherein the robots are configured to regularly produce to the server aregular report under normal operating conditions, wherein the robots arefurther configured to produce to the server a critical condition reportupon occurrence of the critical condition, the server receives, from adetecting robot, a critical condition detected by the detecting robot.Block 410 then transfers control to block 412.

In step 412, the server requests data regarding the critical conditionfrom a robot. For example, the server requests the data from thedetecting robot. For example, the server requests the data from a robotthat is nearest to the critical condition. Block 412 then transferscontrol to block 415.

In step 415, the server displays, on a user interface (UI) visible to auser, the UI operably connected to the server, the critical condition.Block 415 then transfers control to block 420.

In step 420, the server generates a heat map of the critical condition.Block 420 then transfers control to block 425.

In step 425, the server generates a path usable by the robots around thecritical condition. Block 425 then transfers control to block 430.

In step 430, the server switches the detecting robot from periodic,lower-priority data recording, to high-priority, real-time, continuousdata recording and direct uploading. Block 430 then transfers control toblock 435.

In step 435, the server sends the critical condition to one or moreother robots. Block 435 then transfers control to block 440.

In step 440, the server requests assistance for the detecting robot fromthe other robots. Block 440 then transfers control to block 445.

In step 445, the server receives, from the determining robot, adetermination that one or more of the critical condition and a number ofrobots near the critical condition is problematic. Block 445 thentransfers control to block 450.

In step 450, the server sends to one or more other robots a commandblocking the one or more other robots from approaching the criticalcondition. Block 450 then transfers control to block 455.

In step 455, the server receives from a user an instruction regardingpreventive measures to resolve the critical condition. Block 455 thentransfers control to block 460.

In step 460, the server implements the user's instruction regarding thepreventive measures to resolve the critical condition. Block 460 thentransfers control to block 470.

In step 470, the server sends to the UI for display a robot selectionpanel comprising a plurality of robots usable to approach the criticalcondition. Block 470 then transfers control to block 480.

In step 480 the server receives from the UI a user selection of a robotfrom the robot selection panel. Block 480 then transfers control toblock 490.

In step 490, the server transmits to each of the user-selected robots, arespective instruction instructing each of the user-selected robots toapproach the critical condition. Block 490 then terminates the process.

FIGS. 5A-5F are a set of diagrams illustrating a method 500 fordetecting a critical condition by an autonomous mobile robot using asystem for facility monitoring and reporting to improve safety using oneor more robots.

The order of the steps in the method 500 is not constrained to thatshown in FIG. 5 or described in the following discussion. Several of thesteps could occur in a different order without affecting the finalresult.

As shown in FIG. 5A, in step 503, a first robot 110A comprises a firstsensor 220A. A second robot 110B comprises a second sensor 220B. Thefirst robot 110A detects a first human 240A using the first sensor 220A.The first robot 110A also detects the second robot 110B using the firstsensor 220A. The first sensor 220A detects the first human 240A usingthe first robot detection beam 230A. The first sensor 220A detects thesecond robot 110B using the second robot detection beam 230B. The firstrobot 110A determines that a distance 504 between the first human 240Aand the second robot 110B is smaller than is acceptable based on one ormore of user-defined criteria and predefined criteria, creating acritical condition 504. For example, and as depicted, the criticalcondition 504 comprises a near miss between the second robot 110B andthe first human 240A. The first sensor 220A detects the forklift 210using the third robot detection beam 230C. Block 503 then transferscontrol to block 505.

As shown in FIG. 5B, in step 505, a user interface (UI) 175 displays thesame scene that was shown above in three dimensions (3D) in step 503.Visible in the UI 175 are the first robot 110A, the second robot 110B,and a server 506 that operably connects the first robot 110A and thesecond robot 110B. On the UI 175, the critical condition 504 is visible.On the UI 175, visible in outline of the scanned imaging by the firstsensor 220A of the first robot detection beam 230A is the first human240A. On the UI 175, visible in outline of the scanned imaging by thefirst sensor 220A of the second robot detection beam 230B is the secondrobot 110B. On the UI 175, also visible in outline of the scannedimaging by the first sensor 220A of the third robot detection beam 230Cis the forklift 210. Block 505 then transfers control to block 507.

As shown in FIG. 5C, in step 507, a user 508 is viewing on the UI 175the same scene that was shown in step 505. Visible again in the UI 175are the first robot 110A, the second robot 110B, and the server 506 thatoperably connects the first robot 110A and the second robot 110B. Againon the UI 175, visible to the user 508 in outline of the scanned imagingby the first sensor 220A of the first robot detection beam 230A is thefirst human 240A. Again on the UI 175, visible to the user 508 inoutline of the scanned imaging by the first sensor 220A of the thirdrobot detection beam 230B is the forklift 210. Block 507 then transferscontrol to block 509.

As shown in FIG. 5D, in step 509, because of the near miss 504 of thefirst human 240A and the second robot 110B, the first robot 110Aactivates a critical condition for the robots 110A and 110B, to whichthe robots 110A and 110B respond by going into high alert. The UI 175displays the first robot 110A sending to the second robot 110B via theserver 506 its detection of the near miss of the first human 240A andthe second robot 110B. The first robot 110A notifies the server 506 in afirst-robot-to-server transmission 510 of its detection of the criticalcondition. The server declares a critical condition and the UI 175displays a warning flag 514. The server 506 forwards the warning flag514 in a server-to-second-robot instruction 511 to the second robot110B. Shown again are the first robot detection beam 230A, the secondrobot detection beam 230B, and visible in outline of the scanned imagingby the first sensor 220A of the third robot detection beam 230C is theforklift 210. Block 509 then transfers control to block 512.

As shown in FIG. 5E, in step 512, the user 508 is viewing on the UI 175the same scene that was shown in step 505. Visible again in the UI 175are the first robot 110A, the second robot 110B, the critical condition504, the server 506, the first-robot-to-server transmission 510 to theserver 506, and the server-to-second-robot instruction 511. Again on theUI 175, visible to the user 508 in outline of the scanned imaging by thefirst sensor 220A of the first robot detection beam 230A is the firsthuman 240A. Again on the UI 175, visible to the user 508 in outline ofthe scanned imaging by the first sensor 220A of the third robotdetection beam 230B is the forklift 210. Block 512 then transferscontrol to block 515.

As shown in FIG. 5F, in step 515, the server 506 posts the warning flag514 on the second robot 110B in the UI 175. In response to the criticalcondition, one or more of the first robot 110A and the second robot 110Brecord data using their respective sensors 220A and 2220B. In responseto the critical condition, one or more of the first robot 110A and thesecond robot 110B switch from periodic, lower-priority data recording,to high-priority, real-time, continuous data recording and directuploading. The first robot 110A sends the critical condition to otherrobots in a critical area of the critical condition, requesting theiraid in providing further evidence of the critical condition. The firstrobot 110A asks nearby robots to collect sensor data of the environmentaround the near miss 504. The first robot 110A starts recording data andtries to keep the near miss 504 in view by updating its local plan whiledoing the task. The robots 110A, 110B record the data for one or more ofa user-specified time and a predetermined time. The robots 110A, 110Bthen upload the collected data to the cloud. Following completion ofdata collection, a notification is sent to the user 508 in the UI 175allowing the user to do one or more of view the data and decide onpreventive measures to resolve the critical condition 504. Block 515then terminates the process.

FIG. 6 is a flow chart illustrating a method 600 for detecting acritical condition using a system for facility monitoring and reportingto improve safety using one or more robots.

As shown in FIG. 6A, in step 610, a facility 250 depicted in a userinterface 175 comprises three robots, a first robot 110A, a second robot110B, and a third robot 110C are navigating, performing tasks, andwatching for appearance of critical conditions 504. The robots 110A-110Ccomprise respective sensors 220A-220C. The facility 250 furthercomprises the server 506. The facility 250 further comprises a rack 612configured to store goods (not shown). A second robot 110B learns of anearby critical condition 504. The critical condition 504 comprises anoil spill 504. For example, the second robot 110B learns of the oilspill 504 after the second robot 110B slips on the oil spill 504. Block610 then transfers control to block 615.

As shown in FIG. 6B, in step 615, the second robot 110B notifies theserver 606 in a second-robot-to-server transmission 610B of itsdetection of the critical condition 504. Using its sensor 220B, thesecond robot 110B observes the critical condition 504 using a secondrobot detection beam 230B. Also shown again are the first robot 110A,the third robot 110C, the UI 175, the respective sensors 220A-220C, thefacility 250, and the rack 612. Block 615 then transfers control toblock 618.

As shown in FIG. 6C, in step 618, the server 506 sends to the firstrobot 110A a first warning flag 514A in a server-to-first-robotinstruction 511A advising the first robot 110A of the critical condition504. The server also sends to the third robot 110C a third warning flag514C in a server-to-third-robot instruction 511C advising the thirdrobot 110C of the critical condition 504. Also shown again are thesecond robot 110B, the respective sensors 220A-220C, the facility 250,the critical condition 504, and the rack 612. In step 618, the server506 optionally instructs one or more of the first robot 110A and thethird robot 110C to capture data from the critical condition 504. Forexample, the server 506 instructs the first robot 110A to capture animage of the critical condition 504. Also shown again are the secondrobot 110B, the UI 175, the respective sensors 220A-220C, the facility250, and the rack 612. Block 618 then transfers control to block 620.

As shown in FIG. 6D, in step 620, the first robot 110A observes thecritical condition 504 using a first robot detection beam 230A. Thethird robot 110C observes the critical condition 504 using a third robotdetection beam 230C. Also shown again are the UI 175, the facility 250and the rack 612. Block 620 then transfers control to block 625.

As shown in FIG. 6E, in step 625, the first robot 110A uploads to theserver 506 in a first-robot-to-server transmission 510A of itsobservations of the critical condition 504. The third robot 110C uploadsto the server 506 in a third-robot-to-server transmission 510C of itsobservations of the critical condition 504. Also shown again are thesecond robot 110B, the UI 175, the respective sensors 220A-220C, thefacility 250, and the rack 612. Block 625 then transfers control toblock 630.

As shown in FIG. 6F, in step 630, the server 506 merges the data aboutthe critical condition 504 from the first robot 110A, the second robot110B, and the third robot 110C, creating a critical condition warningflag 632. Also shown again are the UI 175, the respective sensors220A-220C, the facility 250, and the rack 612. Block 630 then transferscontrol to block 635.

As shown in FIG. 6G, in step 635, the server 506 uploads the data on thecritical condition 504. The server 506 places the critical conditionwarning flag 532 near the critical condition 504. Also shown again arethe first robot 110A, the second robot 110B, the third robot 110C, therespective sensors 220A-220C, the facility 250, the critical condition504, and the rack 612. Block 635 then transfers control to block 640.

As shown in FIG. 6H, in step 640, the UI 175 displays to the user 508compliance levels with each of five previously designated safety rules641A-641E. Through the UI 175, the user 508 receives a criticalcondition notification 642 from the server (not shown in this figure)via the UI 175 informing the user 508 of the critical condition (notshown in this figure). Using the data presented by the UI 175, the user508 determines a likely cause of the critical condition (not shown inthis figure). Block 640 then transfers control to block 650.

As shown in FIG. 6I, in step 650, the UI 175 accesses data in adatabase. The UI 175 displays to the user 508 a heat map 647 of thecritical condition (not shown in this figure). For example, the heat map647 shows one or more of a location and a frequency of the criticalcondition (not shown in this figure). Also shown again is the rack 612.Block 650 then transfers control to block 655.

As shown in FIG. 6J, in step 655, through the UI 175, the server 506generates a first robot path f for the first robot 110A around thecritical condition 504. Also shown are the second robot 110B, the thirdrobot 110C, and the rack 612. The server 506 may then send the patharound the critical condition 504 to the UI 175 for display. Also shownagain is the rack 612. Block 655 then transfers control to block 660.

As shown in FIG. 6K, in step 660, the UI 175 displays to the user 508the first robot path 657 around the critical condition 504. Also shownagain are the facility 250 and the rack 612. Block 660 then transferscontrol to block 670.

As shown in FIG. 6L, in step 670, the UI 175 displays to the user 508the first safety rule 641A and the second safety rule 641B, and thecritical condition notification 642. The server (not shown in thisfigure) generates a time profile of the critical condition (not shown inthis figure). The server (not shown in this figure) sends the timeprofile to the UI 175 for display. The user highlights the criticalcondition notification 642 and the UI 175 opens the time profile 672 ofa time evolution of the critical condition (not shown in this figure).The time profile 672 provides the user 508 with a time seriesvisualization of the critical condition (not shown in this figure). Thetime series visualization allows the user 508 to see how the criticalcondition (not shown in this figure) evolves across time. The timeprofile 672 comprises a first critical condition time tab 673A, a secondcritical condition time tab 673B, and a third critical condition timetab 673C. The first critical condition time tab 673A comprises a time 9am. The second critical condition time tab comprises a time 12:30 pm.The third critical condition time tab comprises a time 7:45 pm. invitesthe user 508 to request data about the critical condition (not shown inthis figure). The user 508 can select any time tab 673A-673C of interestto learn further information about the evolution at the selected time673A-673C of the critical condition (not shown in this figure). Block670 then transfers control to block 680.

As shown in FIG. 6M, in step 680, the UI 175 displays to the user 508 arobot selection panel 682. The robot selection panel 682 comprises aplurality of robots 110A-110J, the plurality of robots 110A-110J usableto approach the critical condition 504. The robot selection panel 682further comprises a robot checkbox 684A-684J adjacent to each of theplurality of robots 110A-110J. The user 508 selects the first five robotcheckboxes 684A-684E, indicated that the user 508 selects each of thefirst five robots 110A-110E, and further indicating that the user 508does not select any of the second five robots 110F-110J. Block 680 thentransfers control to block 685.

As shown in FIG. 6N, in step 685, the system transmits to each of theuser-selected robots 110A-110E a respective instruction 510A-510Einstructing each of the user-selected robots 110A-110E to approach thecritical condition 504. Block 685 then transfers control to block 690.

As shown in FIG. 6O, in step 690, the UI 175 displays to the user 508 aUI 175 showing each of the user-selected robots 110A-110E as eachapproaches the critical condition 504 following respective robottrajectories 692A-692E.

Now with live, continuous data provided using the UI by the server tothe user for review, the user should be able to determine the culpritbehind the critical condition.

The system may receive from the user, using the UI, additional criticalconditions defined to narrow down a root cause of the emergency. Forexample, the system may receive from the user, using the UI, a requestto notify the user whenever a moving object is detected within aselected area. When this condition gets triggered, all nearby robotswould collect sensor data, providing the system with a more complete anddetailed data set to analyze. Block 690 then terminates the process.

Advantages of embodiments of the invention include that the ability ofthe system to trigger collection of extra data in the case of a nearmiss or another similar emergency allows additional data to be gatheredpertinent to the near miss, data that would not be recorded using aprior art passive recording system. Accordingly, active sensingaccording to embodiments of the invention provides more useful data thandoes passive sensing due to limitations of sensors including one or moreof a limited field-of-view, range limitations, bandwidth limitations,and the like.

Another advantage of embodiments of the invention is that compliance ofthe facility with 5S standards can be promoted by the robotic detectionof a moving object in an area in which that type of object isprohibited.

An additional advantage of embodiments of the invention is that the dataobtained regarding the number of one or more of humans, forklifts, golfcarts, autonomous guided vehicles, autonomous mobile robots and othermoving obstacles in the facility can eventually be used to generatebetter workflows in the facilities to increase one or more of efficiencyand throughput.

A still further advantage of embodiments of the invention is that whenboth the individual robot and one or more robots are streaming data tothe cloud in response to the occurrence of the critical condition, thecloud thereby receives a record of the detected critical condition thatis one or more of multi-dimensional, more complete and more continuous.

A yet additional advantage of embodiments of the invention is that thegenerated statistics and/or heat maps can give the facility betterinformation about flow of material. Another advantage of embodiments ofthe invention is that the generated statistics and/or heat maps can beused to determine inventory areas that are one or more of underutilizedareas and obsolete. Accordingly, embodiments of the invention can beused to optimize facility space utilization.

A still additional advantage of embodiments of the invention is that theinvention can be used to send a robot to do one monitor a specific areato conduct a security audit. An additional advantage of embodiments ofthe invention is that while performing another task, the robot canopportunistically look for one or more of emergency exits, fireextinguishers, and the like. According to yet further advantages ofembodiments of the invention, the robot can also report to the customeron a state of the one or more of emergency exits, fire extinguishers,and the like.

A yet further advantage of embodiments of the invention is that thefiltering settings allow the user to analyze current and historical datain different ways to detect trends and potential trouble spots.

The system and method for facility monitoring and reporting to improvesafety using one or more robots includes a plurality of components suchas one or more of electronic components, hardware components, andcomputer software components. A number of such components can becombined or divided in the system. An example component of the systemincludes a set and/or series of computer instructions written in orimplemented with any of a number of programming languages, as will beappreciated by those skilled in the art.

The system in one example employs one or more computer-readablesignal-bearing media. The computer-readable signal bearing media storesoftware, firmware and/or assembly language for performing one or moreportions of one or more implementations of the invention. Thecomputer-readable signal-bearing medium for the system in one examplecomprises one or more of a magnetic, electrical, optical, biological,and atomic data storage medium. For example, the computer-readablesignal-bearing medium comprises floppy disks, magnetic tapes, CD-ROMs,DVD-ROMs, hard disk drives, downloadable files, files executable “in thecloud,” and electronic memory.

For example, it will be understood by those skilled in the art thatsoftware used by the system and method for facility monitoring andreporting to improve safety using one or more robots may be located inany location in which it may be accessed by the system. It will befurther understood by those of skill in the art that the number ofvariations of the network, location of the software, and the like arevirtually limitless. It is intended, therefore, that the subject matterin the above description shall be interpreted as illustrative and shallnot be interpreted in a limiting sense.

While the above representative embodiments have been described withcertain components in exemplary configurations, it will be understood byone of ordinary skill in the art that other representative embodimentscan be implemented using different configurations and/or differentcomponents.

For example, it will be understood by one of ordinary skill in the artthat the order of certain steps and certain components can be alteredwithout substantially impairing the functioning of the invention.

The representative embodiments and disclosed subject matter, which havebeen described in detail herein, have been presented by way of exampleand illustration and not by way of limitation. It will be understood bythose skilled in the art that various changes may be made in the formand details of the described embodiments resulting in equivalentembodiments that remain within the scope of the invention. It isintended, therefore, that the subject matter in the above descriptionshall be interpreted as illustrative and shall not be interpreted in alimiting sense.

It will be understood by those skilled in the art that software used bythe method for facility monitoring and reporting to improve safety usingone or more robots may be located in any location in which it may beaccessed by the system. It will be further understood by those of skillin the art that the number of variations of the network, the location ofthe software, and the like are virtually limitless.

While the above representative embodiments have been described withcertain components in exemplary configurations, it will be understood byone of ordinary skill in the art that other representative embodimentscan be implemented using different configurations and/or differentcomponents. For example, it will be understood by one of ordinary skillin the art that the order of certain steps and certain components can bealtered without substantially impairing the functioning of theinvention.

The representative embodiments and disclosed subject matter, which havebeen described in detail herein, have been presented by way of exampleand illustration and not by way of limitation. It will be understood bythose skilled in the art that various changes may be made in the formand details of the described embodiments resulting in equivalentembodiments that remain within the scope of the invention. It isintended, therefore, that the subject matter in the above descriptionshall be interpreted as illustrative and shall not be interpreted in alimiting sense.

1-17. (canceled)
 18. A method for detecting for facility monitoring andreporting to improve safety using one or more robots, comprising: usinga system comprising a network, the system further comprising a pluralityof robots operating in a facility, the robots operably connected to thenetwork, the robots configured to monitor facility operation, the robotsfurther configured to detect a critical condition, wherein the robotsare further configured to go into high alert upon detection of thecritical condition, the system further comprising a server operablyconnected to the robots over the network, wherein the robots areconfigured to regularly produce to the server a regular report undernormal operating conditions, wherein the robots are further configuredto produce to the server a critical condition report upon occurrence ofthe critical condition, receiving, by the server, from a detectingrobot, a critical condition detected by the detecting robot; displaying,by the server, on a user interface (UI) visible to a user, the UIoperably connected to the server, the critical condition; switching thedetecting robot, by the server, from periodic, lower-priority datarecording, to high-priority, real-time, continuous data recording anddirect uploading; sending the critical condition, by the server, to oneor more other robots; and requesting assistance, by the server, for thedetecting robot, from the other robots, the method further comprising astep, performed after the requesting assistance step, of: receiving, bythe server, from the determining robot, a determination that one or moreof the critical condition and a number of robots near the criticalcondition is problematic.
 19. A method for detecting for facilitymonitoring and reporting to improve safety using one or more robots,comprising: using a system comprising a network, the system furthercomprising a plurality of robots operating in a facility, the robotsoperably connected to the network, the robots configured to monitorfacility operation, the robots further configured to detect a criticalcondition, wherein the robots are further configured to go into highalert upon detection of the critical condition, the system furthercomprising a server operably connected to the robots over the network,wherein the robots are configured to regularly produce to the server aregular report under normal operating conditions, wherein the robots arefurther configured to produce to the server a critical condition reportupon occurrence of the critical condition, receiving, by the server,from a detecting robot, a critical condition detected by the detectingrobot; displaying, by the server, on a user interface (UI) visible to auser, the UI operably connected to the server, the critical condition;switching the detecting robot, by the server, from periodic,lower-priority data recording, to high-priority, real-time, continuousdata recording and direct uploading; sending the critical condition, bythe server, to one or more other robots; requesting assistance, by theserver, for the detecting robot, from the other robots; receiving, bythe server, from the determining robot, a determination that one or moreof the critical condition and a number of robots near the criticalcondition is problematic; and sending, by the server, to one or moreother robots, a command blocking the one or more other robots fromapproaching the critical condition.
 20. A method for detecting forfacility monitoring and reporting to improve safety using one or morerobots, comprising: using a system comprising a network, the systemfurther comprising a plurality of robots operating in a facility, therobots operably connected to the network, the robots configured tomonitor facility operation, the robots further configured to detect acritical condition, wherein the robots are further configured to go intohigh alert upon detection of the critical condition, the system furthercomprising a server operably connected to the robots over the network,wherein the robots are configured to regularly produce to the server aregular report under normal operating conditions, wherein the robots arefurther configured to produce to the server a critical condition reportupon occurrence of the critical condition, receiving, by the server,from a detecting robot, a critical condition detected by the detectingrobot; displaying, by the server, on a user interface (UI) visible to auser, the UI operably connected to the server, the critical condition;switching the detecting robot, by the server, from periodic,lower-priority data recording, to high-priority, real-time, continuousdata recording and direct uploading; sending the critical condition, bythe server, to one or more other robots; requesting assistance, by theserver, for the detecting robot, from the other robots; receiving, bythe server, from the determining robot, a determination that one or moreof the critical condition and a number of robots near the criticalcondition is problematic; sending, by the server, to one or more otherrobots, a command blocking the one or more other robots from approachingthe critical condition; and receiving, by the server, from a user, aninstruction regarding preventive measures to resolve the criticalcondition.
 21. A method for detecting for facility monitoring andreporting to improve safety using one or more robots, comprising: usinga system comprising a network, the system further comprising a pluralityof robots operating in a facility, the robots operably connected to thenetwork, the robots configured to monitor facility operation, the robotsfurther configured to detect a critical condition, wherein the robotsare further configured to go into high alert upon detection of thecritical condition, the system further comprising a server operablyconnected to the robots over the network, wherein the robots areconfigured to regularly produce to the server a regular report undernormal operating conditions, wherein the robots are further configuredto produce to the server a critical condition report upon occurrence ofthe critical condition, receiving, by the server, from a detectingrobot, a critical condition detected by the detecting robot; displaying,by the server, on a user interface (UI) visible to a user, the UIoperably connected to the server, the critical condition; switching thedetecting robot, by the server, from periodic, lower-priority datarecording, to high-priority, real-time, continuous data recording anddirect uploading; sending the critical condition, by the server, to oneor more other robots; requesting assistance, by the server, for thedetecting robot, from the other robots; receiving, by the server, fromthe determining robot, a determination that one or more of the criticalcondition and a number of robots near the critical condition isproblematic; sending, by the server, to one or more other robots, acommand blocking the one or more other robots from approaching thecritical condition receiving, by the server, from a user, an instructionregarding preventive measures to resolve the critical condition; andimplementing, by the server, the user's instruction regarding thepreventive measures to resolve the critical condition. 22-41. (canceled)42. A method for detecting for facility monitoring and reporting toimprove safety using one or more robots, comprising: using a systemcomprising a network, the system further comprising a plurality ofrobots operating in a facility, the robots operably connected to thenetwork, the robots configured to monitor facility operation, the robotsfurther configured to detect a critical condition, wherein the robotsare further configured to go into high alert upon detection of thecritical condition, the system further comprising a server operablyconnected to the robots over the network, wherein the robots areconfigured to regularly produce to the server a regular report undernormal operating conditions, wherein the robots are further configuredto produce to the server a critical condition report upon occurrence ofthe critical condition, receiving, by the server, from a detectingrobot, a critical condition detected by the detecting robot; requestingdata regarding the critical condition, by the server, from a robot;displaying, by the server, on a user interface (UI) visible to a user,the UI operably connected to the server, the critical condition;generating, by the server, a heat map of the critical condition;generating, by the server, a path usable by the robots around thecritical condition; switching the detecting robot, by the server, fromperiodic, lower-priority data recording, to high-priority, real-time,continuous data recording and direct uploading; sending the criticalcondition, by the server, to one or more other robots; requestingassistance, by the server, for the detecting robot, from the otherrobots; receiving, by the server, from the determining robot, adetermination that one or more of the critical condition and a number ofrobots near the critical condition is problematic; sending, by theserver, to one or more other robots, a command blocking the one or moreother robots from approaching the critical condition; receiving, by theserver, from a user, an instruction regarding preventive measures toresolve the critical condition; implementing, by the server, the user'sinstruction regarding the preventive measures to resolve the criticalcondition; sending, by the server, to the UI, for display to the user, arobot selection panel comprising a plurality of robots usable toapproach the critical condition; receiving, by the server, from the UI,a user selection of a robot from the robot selection panel; andtransmitting, by the server, to each of the user-selected robots, arespective instruction instructing each of the user-selected robots toapproach the critical condition.